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Administrivia 

• Today: Second half of N-Views (n = 2) 

• PS 3: Out tomorrow. 

• Will be due October 12 th . 

• Will be based upon last week and today’s material 
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Two views. . .and two lectures 

• Projective transforms from image to image 


• Some more projective geometry 

• Points and lines and planes 

• Two arbitrary views of the same scene 

• Calibrated - “Essential Matrix” 

• Two uncalibrated cameras “Fundamental Matrix” 

• Gives epipolar lines 
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Last time 

• Projective Transforms: Matrices that provide 
transformations including translations, rotations, similarity, 
affine and finally general (or perspective) projection. 

• When 2D matrices are 3x3; for 3D they are 4x4. 
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Last time: Homographies 

• Provide mapping between images (image planes) taken 
from same center of projection; also mapping between 
any images of a planar surface. 
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Last time: Projective geometry 



A line is a plane of rays through origin 
- all rays (x,y,z) satisfying: ax + by + cz = 0 


in vector notation : 0 = [a b cj 


X 

y 


i p 

A line is also represented as a homogeneous 3-vector I 
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Projective Geometry: lines and points 


2D Lines: ax + by + c = 0 



b c] 


x 

y 

1 


= 0 


Eq of line 


l T x = 0 




P i = 



/ = Pi x p 2 



h = [d b cj 
l 2 - [a 2 b 2 c 2 ] 


x 12 = l 1 xl 
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Motivating the problem: stereo 

• Given two views of a scene (the two cameras not 
necessarily having optical axes) what is the relationship 
between the location of a scene point in one image and its 
location in the other? 
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Stereo correspondence 

• Determine Pixel Correspondence 

• Pairs of points that correspond to same scene point 



Epipolar Constraint 

• Reduces correspondence problem to ID search along conjugate 
epipolar lines 
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Epipolar geometry: terms 

• Baseline: line joining the 
camera centers 

• Epipole: point of intersection 
of baseline with image plane 

• Epipolar plane: plane 
containing baseline and world 
point 

• Epipolar line: intersection of 
epipolar plane with the image 
plane 



All epipolar lines intersect at the epipole 
An epipolar plane intersects the left and right image 
planes in epipolar lines 
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Example: converging cameras 





Figure from Hartley & Zisserman 
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From Geometry to Algebra 

• So far, we have the explanation in terms of 
geometry. 

• Now, how to express the epipolar constraints 
algebraically? 
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Stereo geometry, with calibrated cameras 


X world point 



Main idea 
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Stereo geometry, with calibrated cameras 


X world point 



If the stereo rig is calibrated, we know : 

how to rotate and translate camera reference frame 1 to get to 

camera reference frame 2. 

Rotation: 3x3 matrix R; translation: 3 vector T. 
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Stereo geometry, with calibrated cameras 


X world point 



X, 


If the stereo rig is calibrated, we know : 

how to rotate and translate camera reference frame 1 to get to 
camera reference frame 2. _ tvv _l T 



An aside: cross product 


axb =c 

Vector cross product takes two vectors and 
returns a third vector that’s perpendicular to 
both inputs. 

So here, c is perpendicular to both a and b, 
which means the dot product = 0. 
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From geometry to algebra 

X world point 



X, 


TxX' = 

1 , 1 

Normal to the plane 

= TxRX 


= X' • (T x RX) 


0 = X'-(TxRX) 
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Another aside: Matrix form of cross 
product 



~ 0 


a 2 

V 

axb - 

^3 

0 

~ a i 

b 2 


-a 2 

a l 

0 

h_ 


Can be expressed as a matrix multiplication! ! ! 



Notation: 

axb = [a x ]b 

Has rank 2! 
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From geometry to algebra 

X world point 



X, 


TxX' = 

1 , 1 

Normal to the plane 

= TxRX 
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Essential matrix 


X'-(TxRX) = 0 
X'-([ijRX)=0 


Let E = [T x]R 

X'^EX = 0 


X world point 



E is called the essential matrix, and it relates corresponding 
image points between both cameras, given the rotation and 
translation. 


Note: these points are in each camera coordinate systems. 

We know if we observe a point in one image, its position in other 
image is constrained to lie on line defined by above. 
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Essential matrix example: parallel cameras 




T = 

E = [T X ]R 


r o o (T 
0 06 
0 —B 0 
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Essential matrix example: parallel cameras 

^0 0 0 

E = [T X ]R = o ob 

O-B 0 



IT 



“0 

0 

0 “ 

X 

' , H ' 

• • 

o 

0 

0 

B 

y 


0 

-B 

0 

f 


= 0 


[*' y' f ] 


7y ' 7 V ' 

P' = [X = 


For the parallel cameras, 
image of any point must lie 
on same horizontal line in 
each image plane. 


0 

Bf 

-By 


= 0 
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Weak calibration 

• Want to estimate world geometry without requiring 
calibrated cameras 

• Archival videos (already have the pictures) 

• Photos from multiple unrelated users 

• Dynamic camera system 

• Main idea: 

• Estimate epipolar geometry from a (redundant) set of 
point correspondences between two uncalibrated 
cameras 
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From before: Projection matrix 


• This can be rewritten as a 
matrix product using 
homogeneous coordinates: 

where: 


sx 


im 




im 


= K O 

int ext 












fls x o o x 
0 -f/Sy 0 y 
0 0 1 


Note: Invertible, scale x 
and y, assumes no skew 



CS 4495 Computer Vision - A. Bobick 


Two Views Part 2: Essential 
and Fundamental Matrices 


From before: Projection matrix 


• This can be rewritten as a 
matrix product using 
homogeneous coordinates: 




rx i 



w 

Y w 

s y im 

= K <t> 

int ext 

w 

n 

s 

K 



1 
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Uncalibrated case 


For a given 
camera: 




C 


So, for two cameras (left and right): 


p 


c,left 


= K 


-i 


P 


int,left r im,left 


P 


c, right 


int, right P/m , right 

1 Y / 

Internal calibration 
matrices, one per 
camera 
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Uncalibrated case 

IV' —1 n 

^c, right ^ int, right* im, right 


From before, the essential 
matrix E. 


P ejeft intjeft P imjeft 





P 


int, right im, right 


X 

) int, leftPim, left ) = ® 


P 


im, right 


int, right ) int, left ^P im, 

v y 


left 


= 0 


“Fundamental matrix” F 



or p T Fp' = 0 
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Properties of the Fundamental Matrix 

p r Fp' = 0 


• 1 = Fp' is the epipolar line associated with p’ 

• T = F r p is the epipolar line associated with p 

• Epipoles found by Fp’ = 0 and F T p = 0 (Why?) 

• F is singular (mapping from 2-D point to 1-D 
family so rank 2 - more later) 





Fundamental matrix 

• Relates pixel coordinates in the two views 

• More general form than essential matrix: we remove need 
to know intrinsic parameters 

• If we estimate fundamental matrix from correspondences 
in pixel coordinates, can reconstruct epipolar geometry 
without intrinsic or extrinsic parameters. 
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Any two views? 

• Can we do this for any two views? 

• Yes, as long as can see some similar points between 
images? 


• So how about these... 
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Different Example: forward motion 






CS 4495 Computer Vision - A. Bobick 


Two Views Part 2: Essential 
and Fundamental Matrices 


Computing F from correspondences 


Each point T „ _ n 

correspondence P im, right t Pim, left -U 

generates one 
constraint on F 



/ 11 /12 
/21 /22 
ht f'Z2 




Collect n of these 
constraints 


u\u\ u[v 1 u\ v[ui v[vi v[ Ml Vi 


Solve for f , vector of parameters. 
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The (in)famous “eight-point algorithm” 


(F n \ 

F 12 


250906.36 

183269.57 

921.81 

200931.10 

146766.13 

738.21 

272.19 

198.81 

1.00 

2692.28 

131633.03 

176.27 

6196.73 

302975.59 

405.71 

15.27 

746.79 

1.00 

416374.23 

871684.30 

935.47 

408110.89 

854384.92 

916.90 

445.10 

931.81 

1.00 

191183.60 

171759.40 

410.27 

416435.62 

374125.90 

893.65 

465.99 

418.65 

1.00 

48988.86 

30401.76 

57.89 

298604.57 

185309.58 

352.87 

846.22 

52 S .15 

1.00 

164786.04 

546559.67 

813.17 

1998.37 

6628.15 

9.86 

202.65 

672.14 

1.00 

116407.01 

2727.75 

138.89 

169941.27 

3982.21 

202.77 

838.12 

19.64 

1.00 

135384.58 

75411.13 

198.72 

411350.03 

229127.78 

603.79 

681.28 

379.48 

1.00 



F$2 

\FzZj 


• In principal can solve with 8 points. 

• Better with more - yields homogeneous linear least-squares: 

• Find unit norm vector F yielding smallest residual 

• Remember SVD or substitute a 1? 


What happens when there is noise? 
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Doing the obvious thing 
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Rank of F 



P’ = H ,P =[e'IH,p 

• Let line 1' be the epiloar line _ p 

corresponding to p- goes ~~ P 

through epipole e’ Rank of F is rank of [e’] x - 2 
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Fix the linear solution 


• Use SVD or other method to do linear computation for F 

• Decompose F using SVD (not the same SVD): 

F = UDV t 

• Set the last singular value to zero: 


r 

0 

0~ 


r 

0 

0 ” 

0 

s 

0 

A 

D = 

0 

s 

0 

0 

0 

t 


0 

0 

0 


• Estimate new F from the newD 


✓V A 

F = UDV 
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Stereo image rectification 
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Stereo image rectification 


• Reproject image planes onto a common 
plane parallel to the line between optical centers 

• Pixel motion is horizontal after this transformation 

• Two homographies (3x3 transform), one for each input 
image reprojection 

> C. Loop and Z. Zhang. Computing Rectifying Homographies for Stereo 
Vision . IEEE Conf. Computer Vision and Pattern Recognition, 1999. 
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Rectification 

Example 



(a) Ongmal image 
pair overlayed with 
several epipolar 
lines* 


C. Loop and Z. Zhang, 
Computing Rectifying 

Homographies for 

Stereo Vision, 

IEEE Conf. Computer 
Vision and Pattern 

Recognition, 1999. 



(b) Image pair 
transformed by the 
specialized projec- 
tive mapping II, 
and H' Note that 
the epipolar lines 
are now parallel to 
each other in each 
image* 


(c) Image pair 
transformed by 
the similarity II, 
and H' . Note 
that the image pair 
is now rectified 
(the epipolar lines 
are horizontally 
aligned) 


(d) Final image 
rectification after 
shearing transform 
H and I!'. Note 
that the image pair 
remains rectified, 
blit the horizon- 
tal distortion is 
reduced. 
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Some example cool applications... 
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Photo synth 

Noah Snavely, Steven M. Seitz, Richard Szeliski, " Photo 
tourism: Exploring photo collections in 3D, " SIGGRAPH 
2006 



http://photosynth.net/ 
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Photosvnth.net 


Microsoft • Live Lobs " Q 




Lv- 


Photosynth 



. -1 'a 

Inauguration of the 44th President 

613 photos, 58% synthy 


Sign In | 


Search Synths 




us 



Create your Synth 




Great Pyramid of Giza HDR by mokojolOO 
310 Photos - 95% Synthy 


About Photosynth 








' * 


Explore Synths 


Latest Synth News 


0 ) 

Discussion Forum 


Read the latest Photosynth news and updates on the blog 

Microsoft Corporation | Live Search | MSN | Windov-s Live | Privacy | Legal Microsoft Live Labs | Help | Blog | Contact Us 


Based on Photo Tourism 
by Noah Snavely, Steve Seitz, and Rick Szeliski 
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3D from multiple images 



Building Rome in a Day; Agarwal et al. 2009 
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Summary 

• For 2-views, there is a geometric relationship that define 
the relations between rays in one view to rays in the other 

• Calibrated - Essential matrix 

• Uncalibrated - Fundamental matrix. 

• This relation can be estimated from point 
correspondences - both in calibrated cases and 
uncalibrated. 

• Extensions allow combining multiple views to get more 
geometric information about scenes 

• SLAM (simultaneous localization and mapping) - you’ll hear about 
this if we fit it in... 



